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1. Introduction 

The equilibrium properties of the Hopfield model [|l]], a globally coupled neural network, with 
the typically Hebbian prescription for the interaction strengths 



(in which the represent components of patterns to be stored) have been successfully 
described in the regime close to saturation, where the number p of patterns stored scales as 
p = aN, using replica methods [0, ^. As an alternative approach, a path integral formalism 
developed in [Q], was applied to the dynamics of the same system, and both approaches have 
indeed been shown to lead to identical phase diagrams [Q] . Many modifications of the standard 
Hopfield model have been proposed, including models where the network does not statically 
recall individual patterns, but reproduces a sequence of stored patterns [jl], ^ 0, H, ^, |TOp. 
The simplest way to induce this cyclical behaviour is by an asymmetric modification of 
the interaction matrix (|l]), in combination with a parallel execution of the neural dynamics. 
Numerical simulations show that in such (non-symmetric) models the number of patterns that 
can be stored successfully is significantly larger than that of models storing static patterns, 
with a storage capacity of Uc ~ 0.27 [jl], 0], compared to Uc ~ 0.14 for the standard 
(symmetric) Hopfield model. 

In this paper we study such a model, where a single sequence of extensive length is 
stored in a fully (but non-symmetrically) connected Ising spin neural network with parallel 
stochastic dynamics. The asymmetry of the interaction matrix and the resulting violation 
of detailed balance and associated fluctuation-dissipation theorems rule out equilibrium 
statistical mechanical methods of analysis, including conventional replica theory. Some time 
ago an approximate dynamical solution for this model was proposed [^, which provided 
results roughly in line with the numerical evidence available at the time. To our knowledge, 
an exact solution, however, has so far not yet been found. 

In our present study we use the path integral methods of [Q, ^, |11]] to solve the dynamics 
close to saturation exactly in the thermodynamic limit for our fully connected sequence 
processing network, without having to resort to approximations. In the standard (symmetric) 
Hopfield network two effects limit the storage capacity: a Gaussian noise in the equivalent 
effective single spin problem, which is non-local in time, and a retarded self-interaction. The 
magnitude of both depends on the load factor a. Our theory shows that for the present model 
the retarded self-interaction vanishes, similar to the situation in the non-symmetric SK model 
[|T^ |13], [1^, which explains the extended storage capacity. Numerical simulations for large 
system sizes (up to 50,000 spins) are in excellent agreement with our analytical results, both 
with respect to the maximum storage capacity ~ 0.269 (at zero noise level) and with 
respect to the full phase diagram in the a — T plane. 




(1) 
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2. Definitions 



We study a system consisting of Ising-type neurons cr,; = ±1 which evolve in time 
according to a stochastic alignment to local fields. The neurons change their states 
simultaneously, with probabilities 



Prob[or,(t+l) = -(Ti(t)] 



tanh (3ai{t) 



N 



(2) 



where the entries of the interaction matrix J are given by 



1 ^ 

J = — Vf^'-^V 



(3) 



(the pattern labels fi are understood to be taken modulo p). The non-negative parameter 
P = T^^ controls the amount of noise in the dynamics, with T = corresponding to 
deterministic evolution and with T = oo corresponding to purely random evolution. The 
variables 6i{t) represent external fields. The p vectors = (^f, . . . ,^^) G {—1, 1}^ are 
randomly and independently drawn patterns. Our interest is in the saturation regime p = aN. 
For discussions of the relation of such models to biological or artificial neural networks see 
e.g. [ [T5| , p^ , |T8p. The matrix J will generally be non- symmetric, so that (0) will not 
obey detailed balance. Hence we can not use conventional equilibrium statistical mechanics 
to analyze the stationary behaviour: we will have to solve the dynamics. For the subsequent 
analysis, it will turn out to be useful to represent our expression for J in matrix notation as 



N 



5, 



(4) 



Here the p x N matrix ^ is defined as = . When S is replaced by the unity matrix 1, 
the definition @) reverts to that of the standard Hopfield model. 

To analyze the dynamics of the system we follow [@] and define a generating (or 
characteristic) functional Z[ip]: 



<T{0)...<T{t) 



(5) 



in which cr(s) = (cri(s), . . . , o"Ar(s)) denotes the microscopic system state at time s, and with 
the usual notation x-y = ^ . Xj^/j. In the familiar way one can obtain from Z[ip] all averages 
of interest by differentiation, e.g. 



mi{s) = ((Ti(s)) = i lim f^^] 



Gij{s, s') 



- {ai{s)) = 1 hm 



Cij{s,s') = {ai{s)aj{s')) 



ip^o dipi{s)d6j{s') 

rP^O ^^^{s)^^j{s')' 



(6) 
(7) 
(8) 
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The dynamics is a Markov chain, so the path probabilities p[cr(0), . . . , cr(t)] are 
simply given by products of the individual transition probabilities iy[cr'|cr] of the chain: 
p[cr(0), . . . , cr(t)] = p[cr(0)] nl=I) W[(t{s + l)|cr(s)]. For the dynamics (0) these transition 
probabilities are given by 



W[a{s+l)\cT{s)] = \[ \ 



1 + (Ti(s+1) tanh(/?[^ J,jaj{t) + ei{t)]) 
j 

^ n ^ /5'^»(^+l)Ej J»j<Tj(s)+6»,(s)]-ln2cosh(/3Ej J^Ja,is)+e,(s)]) 
i 

To formally remove the coupling terms ai{s + l)aj{s) we introduce the auxiliary variables 
h(s) = (/ii(s), . . . , /lAr(s)), representing the local fields at each spin site at given times, by 
insertion of 

1= f dh{s)l[5Ms) -J2-JrM'^) - 

i j 

After writing the above (5-distributions in integral form, which generates conjugate field 
variables h(s) = {hi{s), . . . ,hN{s)), and upon introducting the more convenient notation 
{dhdh} = Yli ns<tM^«('5)'^^i('5)/27r], we can express (|]) as 



/3o-(s+l)-h(s)-J].ln2cosh[/3/ii(s)]+ih(s)-[h(s)-0(s)]-ii/>(s)-(T(s) 

cr{0)...cr{t) s<t 



This expression describes the system dynamics (g,^ in general form. To obtain quantitative 
information about particular regimes of operation, we have to make specific ansatze. Our 
ansatz will be one describing (possibly noisy) recall of the stored sequence of patterns. At each 
time step exactly one stored pattern is assumed to be 'condensed' , i.e. the overlap between that 
pattern (which without loss of generality can be labelled with the time index) and the system 
state is of order 0{1), whereas all other overlaps are of order O ( 2 ) . The cumulative impact 
of the overlaps of the non-condensed patterns will introduce an additional noise component 
into the system dynamics; the non-condensed patterns play the role of 'quenched disorder'. 
For N ^ 00 the mean-field physics of the problem should be self-averaging with respect to 
the realisation of the disorder, so we are allowed to average the generating functional (^ over 
the non-condensed patterns (such averages will be denoted as /[{^}]). Since each pattern 
with fi < t will at some stage be condensed, in contrast to those patterns with fi > t,we can 
simplify our calculation by averaging only over the latter. The resulting expressions will, for 
^ 00, turn out not to depend on the remaining patterns with n < t. 
As in most dynamic mean-field calculations of disordered systems based on evaluating 
disorder- averaged generating functionals, we will consider the time t to be fixed, whereas 
we will take the limit ^ 00. This restricts the predicting power of the theory to those 
processes that take place on finite time- scales. In the present calculation we will find that 
a time-translation invariant state (representing motion on a stationary limit-cycle) is indeed 
approached on finite time-scales, so this restriction is not a problem. 
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In (Q) only the term h(s)-(^^S^)cr(s) contains both condensed and non-condensed patterns. 
We isolate the non-condensed ones by introducing the variables x and y: 



dxnn* 



i=/dynn* 

s<t 



Upon writing the above 5-distributions in integral form (which generates the additional 
integration variables x and y), we then arrive at the following expression for the disorder- 
averaged generating functional: 



ln2cosh[/3fcj(s)]+ih(s)-[h(s)-0(s)]-ii/'{s)-<T{s) 



<T(0)...<T(t) 



X e 



-iN- 



'E«<t[h{^)-r+^][<T(s).r] / dx dx dy dy ^ \Y.s<tT.^^s[^^{s)^^{s)+v^{s)y, 



(27r)2{p-i)t 



(10) 



We can now carry out the disorder average in the last term, which is significantly simplified if 
in the exponent we use E^^JE* • • • ] = E^>JEi • • • ] + 0{N). It gives 



-^^N-^Y.s<tT.^,>tT.^li':Y^^^~l{s)h,{s)+y^,{s)a,{s 



nn 



COS 



s<t 



Q 0{N2) ^^-^Y.s,s'<tY.r[S^^^-l{s)k{3)+y^{s)a,{s)\[x^.^{3'^^^^^ 

tM>t 



(11) 



Since the leading order in the exponent of ([TT]) does not involve components of {x, x, y, y} 
with pattern index fi < t, the latter can be integrated out in expression ([T0|). We now isolate 
the various relevant macroscopic observables occurring in ( [TTt ) by inserting integrals over 
appropriate 5-functions: 



(27r/iV)* 
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J i2n/Nr 
J i27r /Nr 
J i2n/Nr 

Combination of ( [TT| ) with ( [T0| ) will then give us an expression for Z[il^] which will factorise 
over sites if we choose a factorised initial distribution p(cr(0)) = YliPii'^iW)^ resulting in an 
integral which for ^ oo will be dominated by saddle-points: 

Zl-iP] = y"dm dm dk die dq dq dQ dQ dK dK e NW-]+'S>l-]+n[...]}+o{Nh (^2) 

in which the functions \E'[. ..],$[... ] and . . ] are given by: 

\l/[m, k, m, k, q, Q, K, q, Q, K] = i m(s)m(s) + k{s)k{s) — m{s)k{s) 



s<t 



+ i s')q{s, s') + Q{s, s')Q{s, s') + K{s, s')K{s, s') 



(13) 



s,s'<t 



$[m,k,q,Q,K] = ^J]lnJ p,(a(0)) /"{dhdh} e ^-'[^'^ 



(s+l)/i{s)-ln2cosh[/3ft.(s) 



{s)+yt,{s)yf,{s)-Xf,{s)y^{s)\ 



X e "5 SM>t Es,s'<J^M(s)Q(«.sO^M(«0+^M-l(s)'^(s^s)^'M(«0+^'M(s)-?i■(s,sO£M-l(sO+SM(«)g(s,sOSM(«'' 

N J (27r)(P-*)* 

X e "5 SM>t Es,s'<t['"M(*)Q(«'*')"M(s')+«M{^*)-f*^(«'.«)»'M{«')+'"M(s)'^(s.«')a:M(s')+'JM(s)q{s.s')'^M{«')] 

(15) 



with the short-hand {dhdh} = Yls^t[dh{s)dh{s) /2tt]. The final expression ( [T5| ) for $ was 
obtained by integrating out the variables (x, y), followed by a simple pattern index shift 
transformation. 
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One can deduce the physical meaning of the various dynamic order parameters 
introduced along the way in the usual manner by (repeated) derivation of the definition 
(^ with respect to the external fields 6i{s) and il)i{s), in combination with usage of the 
normalisation identity Z\0\ = 1. Evaluation of a function /[...] at the dominating (physical) 
saddle-point of the extensive exponent in ( }T^ will be indicated by /| saddle- The external fields 
occur in the function $ only (not in \1' or fi). The resulting identities can be summarised in a 
compact form upon introduction of an effective single-site measure (. . .)■, defined as 



f[{a,h,h}] 



E.(o)....w /{dhdh} W,[{a, K h]] /[{a, K h]] 



E.(o)....w/{dhdh}W^4{a,/.,M] 



with 



E.<t[/3<^{^+l)'i{«)-ln2cosh[/3h{s)]+ih{s)[/i{s)-e,(s)-fc{s)?^''+i]-i(7(s)m(s)C|] 



X e 



-iEs s'<tm^'^>(s)^(^')+Q(S'^')Ks)Ks')+K(s,s')a(s)h{s') 



saddle 



In particular we now find, in leading order in N: 



(o-i(s)) = i lim 



. dim 



•0^0 dipi{s) 



saddle 



99,(s) *-o d0t{3) 



saddle 



i(h{s] 



(16) 



(17) 



(18) 



{ai{s)aj{s)) 



lim 

^-^0 diJi{s)dijj{s') 



saddle 



lim 



saddle 



(19) 



— 1 lim 



d9j{s' 



V-o dilJi{s)dej{s') 



saddle 



+ i lim 



diJiis) delis') 



saddle 



i6ij ((r{s)h{s'] 



(20) 



d^Z[0] 



lim 



92(iV$) 



dei{s)dej{s') v-o de,{s)dej{s' 



saddle 



+ lim 



saddle 



= -6ij (^h{s)h{s') ^ 
Note that we have already used identity ([T^ to simplify ( PD| ) and (pi]). 



(21) 
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4. Derivation of Saddle Point Equations 



In the limit ^ oo, the integral ( [T2| ) will be dominated by the dominating (physical) 
saddle point of the extensive exponent \E'+$+r2. We are now in a position to derive 
the saddle-point equations by differentiation with respect to our integration variables 
{m, k, m, k, q, Q, K, q, Q, K}. These equations will involve the average sequence overlap 
m{s) (which measures the quality of the sequence recall) and the average single-site 
correlation- and response functions C(s, s') and G{s, s'): 



N—^00 I\ ' 

i 

N^oo iV •^"^ 



Gis,s') 



N^ooN^ dOAs') 



(22) 



(23) 



(24) 



Straighforward differentiation, followed by usage of the identities ( p^pj ) wherever possible, 
then leads us to the following saddle-point equations: 



for all s, s' 



for all s : 



k{s) 



m[s] 



Qis, s') = 



i 

for all s, s' : q{s, s') = C{s, s') = lim — (cr(s)cr(s')) ■ 

i 

for all s, s' : K{s, s') = iG{s, s') = lim — (^a{s)h{s'] 
for all s, s' : q{s, s') 



for all s, s' 



for all s, s' 



Q{s,s') 
k{s,s') 



dq{s, s') 



saddle 



dQis,s') 



dK{s, s') 



saddle 



saddle 



(25) 
(26) 
(27) 
(28) 
(29) 
(30) 
(31) 



The effective single-site measure ( [T^ ) simplifies considerably due to ( P5|P^ , and since the 
function depends on the trio {q, Q, K} only (see (|T5|)), our saddle-point equations can be 
reduced to a problem involving only the key physical observables m(s), C{s, s') and G{s, s'). 

In order to calculate the remaining Gaussian integral in ^7 we have to define matrices 
operating in the product space of vectors labelled by both time indices s and pattern indices 
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/i. Note: in the case of the standard symmetric Hopfield model, where integration variables 
with different pattern labels /i immediately decouple, this would not have been necessary. 
We define a matrix F = S R as having matrix elements r^^/(s, s') = S^^iR{s, s'), where 
fi, fi' = 1, . . . ,p and where s,s' = 0, . . . ,t— 1. It will operate as follows: if y = Fx 
then yij.{s) = J2fi'>t J2s'<t S^lfJ.'Ris, s')Xf^i{s') for each (yU, s). Note that in evaluating for 
N ^ oo, and due to t remaining finite, we can safely drop the restriction that fi > t, and 
instead have /x = 1, . . . ,p. The above definition allows us to write 

lim fi[C,Q,iG]= lim ^In /"^li^^ g -i^-l^^Ql^-^ li^^'^+^^-l^^i-^^^l" 

= lim iln|det-^[l®C] e '^"■{^^Q+t^^^-^^^l'I^^^l-^I^^^-^^^l}"! 

JV-ooA^ [ ^ ^ J (27r)P*/2 J 

Jim In det[l(g)C] +ln det {l®Q+[S®l-l®G]"f[l(g)C]"^[S(g)l-l(g)G]} 



(32) 



We use to work out the saddle-point equations (g9|-|T])- The first of the three equations 
comes out trivially: 

^(.,.')-i^^[C,0,iG] 

^ ~ n'^oo^ dCis s') In det{[S®l-l®G]t[S®l-l®G]} =0 (33) 

In order to work out the remaining two equations we use the general matrix identity 
lndet[M+Q] = IndetM + Tr[M~iQ] + 0{Q'^), as well as the specific properties of the 
p X p matrix = In particular we will be using its unitarity, S^S = 1, the identity 

[(S^^)'"S"]^^ = 5mn, and its p eigenvalues being given by = e'^'^'^/P. Equation ( pO| ) 
now reduces to 

= - lim {[S®l-l®G]t[l®C]-^[S®l-l®G]}"'(s',s) 
= - lim {[l®l-St®G]-i[l®C][l®l-S®Gt]-n (s',s) 



E ^liinV{[St®Gr[l®C][S®Gtr}^^ 

n,m>0 M5~P 
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giving: 



Q = -im5^(Gt)"C(Gr (34) 

n>0 

Finally we turn to equation (|3T|): 

1 d 1 

= — « T lim - |ln det[l(8)l-S"^®Gl"^ + In det[l®l-S"^®Gl| 

2 oG[s, s') p-"x> p ' 



1 d 
--a 



- lim -Vjln det[l-e-2"'^/PG^] +ln detil-e^^'^/^G]} 



2 SGfs, s') p^oo p 

fi<p 



5. The Effective Single-Spin Problem 

Let us summarize the present stage of our calculation. Most macroscopic integration 
variables are found to vanish in the relevant physical saddle-point: k{s) = rh(s) = 
Q{s, s') = q{s, s') = K{s, s') = 0. The remaining ones can all be expressed in terms of 
three macroscopic observables, viz. the overlaps m{s) and the single-site correlation and 
response functions C{s, s') and G{s, s'), as defined in ( ^J23| , ^4] ), by using the four equations 



). We are thus left with a set of closed equations (^^^ from which to solve 
{m{s), C{s, s'), G{s, s')}. These equations are defined in terms of an effective single-spin 
problem. At this stage it is natural to choose the remaining external fields 9i(s) to be so-called 
'staggered' ones, i.e. 9i{s) = 9{s)$,^~^^. If used as symmetry-breaking perturbations, such 
fields will exactly single out macroscopic solutions of the type we introduced as an ansatz. 
This choice also removes the formal need to break symmetries via initial conditions, so that 
we may now choose pi{a{0)) = p(cr(0)). As a consequence we find that the single-site 
measure (|16|) becomes site-independent, since the remaining site dependence due to pattern 
components can be eliminated via a gauge transformation whereby a{s) cris)Q and 
h(s) h(s)^^'^^. The resulting single-spin problem involves the following measure (which 
is properly normalized, as can be verified by explicit evaluation of (1)^): 



(/[W]).= E /{dhdh}pKO))/[M]e 



E,<t[/3<7(''+l)^(s)-ln2cosh[/3/i(s)]] 

a(0)...a{t) 



with R{s, s') = X]n>o[(^^)"^(^)"]("^' ^')- 'This measure describes an effective single spin 
cr(s) with a stochastic alignment to local fields given by /i(s) = m(s)+6'(s)+Q;50(s), in which 
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the term represents a zero-average Gaussian random field with (non-zero) temporal 
correlations (0(s)0(,s')) = R{s,s'). Note that, as a consequence of (P5|), there is no term 
representing a retarded self-interaction, in contrast with the standard (symmetric) Hopfield 
model. This is the mathematical explanation of the higher storage capacity in the present 
sequence processing model. The asymmetry of the interaction matrix prevents the build-up of 
a microscopic memory, similar to the situation in the non-symmetric SK-model [|T^ |T3|, |T^. 
The equations from which to solve our remaining order parameters can be written as 

m{s) = {a{s))^ (37) 



C{s,s') = {ais)ais'))^ (38) 

G(.,^') = ^M^)). (39) 

Since the measure ( |36| ) factorizes with respect to spin variables at different times, we can 
immediately perform the spin summations in ( |37|j38|j39| ) (which would not have been possible 
for the standard Hopfield model). After a simple rescaling of fields and conjugate fields we 
then arrive at 

m(s) = J{dvdw}e'''-^-'^'"-'^'^ tanhp[m{s-l)+e{s-l)+ah{s-l)] (40) 

C{s,s') = 6s,s' + [l-Ss,s'] y"{dvdw} e'^-^-5^-^w tanh p[m{s-l)+9{s-l)+a'^v{s-l)] 
X tanhp[m{s'-l)+9{s'-l)+ah{s'-l)] (41) 

s') = P5s,s'+i 1 1 -y"{dvdw} e'^-^-i^-^^ tanh^ p[m{s-l)+e{s-l)+Jvis-l)] | 

(42) 

with R{s, s') = X]n>o[(^^)"^(^)"]('^' response function is found to be non-zero 

only if field perturbation and spin measurement are temporally separated by exactly one 
iteration step. Thus anomalous response cannot occur, and macroscopic stationarity should 
be achieved on finite time-scales. 

6. The Stationary State 

We now choose stationary external fields 9i(s) = 9^^~^^, giving 9{s) = 6' in terms of the single- 
spin problem, and inspect time-translation invariant solutions of our macroscopic equations 
( p0tpT1 , p2| ), which will describe motion on a macroscopic limit cycle: 



m{s)=m Cis,s') = C{s-s') G{s, s') = G{s - s') 



(43) 
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In order to do this we shift the initial time in from = to to = — oo, and the 



final time to t = cxo. According to (^3]) the matrices C and G become Toeplitz matrices and 
commute, which implies that the matrix R simplifies to 

R = C [1 - G^G] , R{s, s') = R{s-s') (44) 



and that we may thus write the stationary version of ( |4-0| , |4-1| , |4-2| ) as 
m = y"{dvdw} e^v-w-iwRw tanh/5[m+^+a5t;(0)] 

C(r^O) = y"{dvdw} e'^-^-^^-^^ tanh P[m+9+a^ v{t)] tanh P[m+9+a-^v{0)] 

G{t) = p6r,i |l -j{dYdw} ^w-Rw tanh^ p[m+9+ahi0)]j 

We separate in C(r) and R{t) the persistent from the non-persistent parts, i.e. 

C{T) = q + C{T), R{T)=r + R{T), lim C'(r) = Um ^(r) = 

T— >itoo r— >itoo 

The persistent part r of R{t) can be expressed in terms of the persistent part q of C(r), by 
combining ( P^ ) with the above expression for G{t) . This separation of persistent parts induces 
a frozen random field into the above order parameter equations, which can subsequently be 
absorbed into the local fields: 



(with the familiar abbreviation Dz = (27r) ae 2^ ). Upon rewriting G{t) = (36r^i [1 — q] 
and r = qp, we arrive at the following expressions for our persistent observables: 

m = Jdz y"{dvdw} e'^-^-^^-^^ tanhp[m+e+z^/aqp+ah{Q)] 

q= lim Dz /{dvdw} e^^'^"^^'^^ tanh/5[m+^+;2v^+«^^(^)] 

X tanh P[m+6+z^/aqp+a^v{0)] 

q = joz j{d^Jdw}(^^■^-^^^■^^ ianli^ (3[m+e+z^^+a^v{Q)] 



p = [i~f3\i - qr] 

We only need to know the joint probability distribution of the pair (t>(r), t>(0)) in the limit 
r — s> 00 to work out the remaining integrals over v and w. This distribution is clearly a 
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zero-average Gaussian one, so finding the second order moments suffices. Integration over w 
gives 



MrMO)) = [detR]-i /n 



dt>(s) 
2^ 



e 2 



v{t)v{Q) = R{r) 



from which we conclude that {v{OY) = R{0), and that limT-^oo {v{t)v{0)) = 0. The variance 
^(0) = R{0)-r immediately follows from (p4|): 



^(0) 



1-q 



1 -/32(l-g)2 

All remaining integrals are now expressed in terms of persistent observables only: 



m 



j Dz J Dx tanh/5[m+^+2;yagp+XA/a(l— 



q= Dz 



Dx tanh P[m+9+z^/aqp+x ^/a{l—q)p] 



If we finally combine the two Gaussian variables in the equations for m and q into a single 
Gaussian variable we arrive at our final result: 



p=[l-P\l-~qfr" 



(45) 



m 



Dz t3Jl\l[3[m+6+Z^/Cif>\ 



^ ~ J tanh"^ P[m+9+Zy/ap\ 

g = y Dz J Dx tanh/?[m+6'+zyagp+XA/a(l--g)p] 



(46) 



(47) 



(48) 



Note that the trio ( P3|J^ , P7| ) form itself a closed set, from the solution of which the persistent 
correlation q simply follows. 



7. Phase Diagram and Storage Capacity 



We have solved the coupled equations (p3|-p7|) numerically for 6* = | in order to determine 
the region in the a-T plane where solutions with m ^ 0, which describe pattern sequence 

I The alternative choice 6* 7^ would have described the less interesting scenario where the m 7^ state would 
not be sustained autonomously (if at all), but where at each time step and at each site a very specific external 
field 9i{s) = 6S^^^^ would have actively pushed the system towards the pattern sequence. 
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Figure 1. Phase diagram of the sequence processing network, in which one finds two phases: a recall 
phase (R), characterized by {m 7^ 0, (7 > 0, q > 0}, and a paramagnetic phase (P), characterized 
by {m = 0, g = 0, g > 0}. Solid line: the theoretical prediction for the phase boundary. Markers: 
simulation results for systems of TV = 10, 000 spins measured after 2, 500 iteration steps. The precision in 
terms of a is at least Aa = 0.005 (indicated by error bars); the values for T are exact. 



recall, exists. The boundary of this region determines the storage capacity of the system. 
This theoretical result was tested against numerical simulations of the present model, carried 
out at the spin level @. We show the results in figure [1]. One finds that, for T > and 
a < 00, the equations (P5|-P7|) admit only two types of solutions: a recall solution (R) 
characterized by {m 7^ 0, g > 0, g > 0}, and a paramagnetic solution (P) characterized 
by {m = 0, g = 0, g>0}. The absence of the analogon of a spin-glass phase will be 
discussed in more detail below. The phase boundary R^P as obtained theoretically (solid 
line) shows an excellent agreement with the computer simulations (markers), as performed 
for systems of size = 10, 000 (using bi-section). The maximum storage capacity is 
obtained in the zero noise limit T — > (or /3 ^ 00). For /? 00, where g — > 1 and g ^ 1, 
the saddle point equations can be simplified in the usual manner, using identities such as 



lim / tanh /3 [m + z^fap\ = erf 

(3^^ J V 



lim — g) 



d 



dm 13 



lim / Dz tanh dim + zJ'ap\ 



Trap 



exp 



m 



2ap 



With the definition x = m/y/2ap, from which the overlap m follows according to m = 
erf (x), we can combine our saddle point equations for /3 ^ 00 into the single transcendental 
equation 



x\/2a = ±\ eif^ix 



4x2 



TT 



exp (— 2a;2 



(49) 
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Figure 2. Results of determining the maximum sequence storage capacity etc at T = via numerical 
simulation of networks with different sizes N . The values for have been determined with a precision of 
at least Aa = 0.001 where possible. Note that the N = oo theory predicts ac « 0.269. 



This equation is identical to that obtained in the T = limit for the layered model of [[T9|], and 
for the present model we thus obtain the same maximum storage capacity, which is defined as 
the largest value of a for which (^9]) has non-trivial solutions, of etc ~ 0.269. Note, however, 
that this equivalence does not extend beyond the T = limit. To also verify this latter 
result with numerical simulations, taking into account the possibility of finite size effects, we 
measured the maximum storage capacity in zero temperature simulations for different system 
sizes, ranging from N = 2, 500 to = 50, 000. This resulted in figure ||. The numerical data 
are again perfectly consistent with the result ac ~ 0.269 of our N = oo theory. 

Finally we turn to the non-recall phases, still for zero external field, where m = and 
where the remaining order parameters q E [0,1] and q E [0,1] follow from solving the coupled 
equations 



Dz tanh 



(3z 



a 



1 -/32(1 -g)2 



Bzl / Dx tanh p 



aq 



l-/52(l-g)2 



+ X\ 



(50) 



(51) 



The first of these equations ( pO| ) determines q, which is related to the response function 
via G{t) = P{1 — q)5r,i. Its solution is unique. For finite temperature one finds that q is 
always non-zero, approaching zero only asymptotically as q = ajS"^ + O {(3^) for T ^ oo. 
The persistent correlation q subsequently follows from solving (pi]). This second equation 
always admits the paramagnetic solution g = 0. Careful numerical and analytical inspection 
reveals that for T > and « < oo it admits no solutions with g > 0, which would have 
been the analogon of a spin-glass state. Only in the limits T — > and a ^ oo, where 
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P{1 — q) (1 + |7ra)^5 and P{1 — q) 1 respectively, and where equation ( pTj ) converts 
into 



erf^ 



does one find a non-trivial solution, namely q = I. This implies that in the phase diagram of 
figure lU the phase beyond the boundary of the recall region is a paramagnetic state, with only 
a transition to a spin-glass type frozen state precisely at T = 0. This type of behaviour is very 



similar to that observed in non-symmetric spin-glass models [{TJ, [Tj, [IJ]. 
8. Discussion 

In this paper we have used path integral methods to solve in the thermodynamic limit the 
dynamics of a non-symmetric neural network model, designed to store and recall sequences 
of stored patterns, close to saturation. For about a decade this model has been known from 
numerical simulations to have a significantly enlarged storage capacity (by about a factor 
two) compared to the more familiar symmetric Hop field network [|l], 0, |]], which stores static 
patterns and obeys detailed balance. So far the sequence processing model had not yet been 
solved, and thus the enlarged storage capacity had not yet been explained, mainly due to the 
complication that the absence of detailed balance rules out the more traditional equilibrium 
statistical mechanical methods, including replica theory. In contrast, even in the regime of 
interest where the number of patterns in the sequence scales as p = aN, and where thus the 
dynamical methods of simple mean-field models cannot be used, the powerful path integral 
methods of [TT], ^ do still apply; they allow us perform the disorder average in a dynamical 
framework, and thereby to calculate the system's phase diagram without having to resort to 
additional approximations. 

In the standard (symmetric) Hopfield network two effects limit the storage capacity: a 
Gaussian noise in the equivalent effective single spin problem, which is non-local in time, and 
a retarded self- interaction. The magnitude of both contributions depends on the load factor a. 
For the present model we find, in contrast, that the retarded self-interaction vanishes, which 
explains the extended storage capacity. Numerical simulations for large system sizes (up to 
= 50, 000 spins) are in excellent agreement with our analytical results, both with respect 
to the maximum storage capacity ~ 0.269 (at zero noise level) and with respect to the full 
phase diagram in the a — T plane. In the limit of zero noise level we find that the equation 
from which to solve the order parameter which describes the quality of the sequence recall 
reduces to that of the layered model of [ fT^ ] . Our order parameter equations and their solutions 
also turn out to be very similar to those found for various versions of the non-symmetric SK 
spin-glass model, as studied in [IT^, 0|, |T^. In particular, common features are the absence of 



a retarded self-interaction in the effective single-spin problem, and the absence of a spin-glass 
type phase for non-zero temperatures. 

As a next step one could apply the present formalism to networks which store more than 
one periodic pattern sequence. By varying the scaling with N of both the sequence length 
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and of the number of sequences, one should expect a transition between the behaviour similar 
to that of the symmetric Hopfield model (with an effective retarded self-interaction) and the 
behaviour observed in the present model (without such a retarded self-interaction). This will 
be the subject of a future study. 
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